Instance-aware Prompt Learning for Language Understanding and Generation

Prompt learning has emerged as a new paradigm for leveraging pre-trained language models (PLMs) and shown promising results in downstream tasks with only slight increase parameters. However, the current usage of fixed prompts, whether discrete or continuous, assumes that all samples within task share same prompt. This assumption may not hold diverse require different prompt information. To address this issue, we propose an instance-aware method learns each instance. Specifically, suppose learnable token contribution to instances, learn by calculating relevance score between instance token. The weighted would be aware. We apply our both unidirectional bidirectional PLMs on understanding generation tasks. Extensive experiments demonstrate achieves comparable using few 1.5% parameters tuned obtains considerable improvements compared strong baselines. In particular, state-of-the-art ALBERT-xxlarge-v2 SuperGLUE few-shot benchmark.